Real-time, low memory estimation of unique client computers communicating with a server computer

ABSTRACT

Computer systems and methods for estimating the number of client computers actively coupled to a server computer system in real-time are discussed herein. Specifically, systems and methods are discussed for a server computer system receiving heartbeat messages from a plurality of client computers and generating an estimate of the number of client computers actively coupled to the server computer system in real-time without locks, such as a database table lock. A heartbeat message from a client computer need not include a client or user identifier. In an embodiment, the memory footprint/overhead is O(l), and may be a single whole number greater than zero, such as a 64-bit unsigned integer. Systems and methods are also discussed herein to calculate and reduce the expected error of the estimated number of active clients.

CROSS-REFERENCE TO RELATED APPLICATIONS; BENEFIT CLAIM

This application claims the benefit of U.S. Provisional Application No.62/050,684, filed Sep. 15, 2014, and U.S. Provisional Application No.62/191,977, filed Jul. 13, 2015. Both provisional applications areincorporated by reference herein.

FIELD OF THE INVENTION

The present invention generally relates to server-side analytics andmore specifically to determining the number client computers that areactively transferring data with one or more server computers. SUGGESTEDGROUP ART UNIT: 2447 (ELECTRICAL COMPUTERS AND DIGITAL PROCESSINGSYSTEMS: MULTICOMPUTER DATA TRANSFERRING); SUGGESTED CLASSIFICATION:709.

BACKGROUND

A website analytics system can collect telemetry data for a website. Thetelemetry data may identify which webpages or other content have beenrequested or downloaded and by which client computers. For example, awebsite analytics system can generate various statistics about awebsite, such as the number of times a particular webpage of the websitewas served, the number of unique visitors the website received each dayover the past month, where the visitors of were geographically located,or what browser each client computer was using. Website administratorsmay use the statistics to gauge how much advertisers should pay perimpression, determine which web pages are most popular, or determinewhat types of client devices are requesting data or webpages. Websiteadministrators may also use the statistics to determine how to allocateresources, such as hardware or developer time. For example, if theserver computers for a website are taking more than a particular amountof time to serve a particular number of webpage impressions, then awebsite administrator may instantiate additional server computers toreduce the amount of time the website takes serve the same number ofwebpage impressions.

A website analytics system may generate and receive large amounts oftelemetry data for a website. Due, at least in part, to the volume ofinformation generated, the website analytics system may take a long timeto process the telemetry data to generate one or more statistics.Accordingly, a website analytics system may dedicate a first set ofcomputers or processes to receive and store telemetry data, and a secondset of computers or processes to generate one or more statistics fromthe telemetry data “offline”. Generating a statistic offline meansgenerating the statistic after the telemetry data used to generate thestatistic is received, aggregated or combined with other telemetry data,and stored in persistent storage for subsequently requested reports. Incontrast, generating a statistic in “real-time” means generating thestatistic while the telemetry data affecting the statistic is beingreceived, or shortly thereafter.

Some statistics may be more useful if generated and acted upon inreal-time rather than offline. For purposes of illustrating a clearexample, assume a website comprises a set of server computers in a cloudcomputing infrastructure, where the set of server computers areconfigured to support up to 100 different active client computersconcurrently streaming video. Assume further that the websiteadministrator may allocate or deallocate cloud resources to the websiteat will. If more than 100 client computers are actively trying to streamvideo from the website, then the website may be overwhelmed and theclient computers may have intermittent pausing or longer than expecteddownloading times. The website administrator may not realize the currentnumber of server computers allocated to streaming video is insufficientuntil users begin to complain or a latent offline statistic or report isgenerated. In contrast, if a website administrator could determine inreal-time how many client devices are currently streaming video, thenthe website administrator could allocate or deallocate resources, suchas cloud server computers, as needed. However, determining a statistic,such as how many client devices are active or currently downloadingvideo, may be difficult.

One way to determine how many client devices are actively communicatingwith a webserver is to maintain a database table of active devicescomprising device identifiers that identify which devices have requesteddata within a particular amount of time. For example, each time a clientcomputer requests data from a server computer, a tracker process maystore an identifier of the client computer and a current timestamp in arecord of the database table. An identifier may be the IP address or MACaddress of the client computer or a username of a user using the clientcomputer. If a request is received from an identifier already in arecord in the database table, then the tracker process update thetimestamp in the record to correspond to the current time. One or morereaper processes may periodically review the list of addresses andremove addresses from the table that do not have a timestamp thatcorresponds to a time that is within a particular range of the currenttime. One or more aggregation processes may periodically count thenumber of addresses are currently in the database table to determine howmany client computers are connected to the webserver. The one or moreaggregation processes may store the counts in a database for futurereports generated offline.

A system that implements the method discussed above can require amassive amount of computational and engineering resources. For example,to make sure a one process, such as a tracker process, does not try toadd or update an entry, while a second process, such as a reaper processor aggregation process, scans or deletes an entry from the database, thedatabase may impose locks on the database table. Locking may quicklyslow down performance of the database to the point where the aggregationprocesses are no longer accurately counting how many clients areactively communicating with the webserver. Accordingly, a website maymaintain the database table as one or more shards distributed across oneor more database server computers, which may allow a process to lock ashard or row in the share without locking the portions of the table inother shards. The website may maintain a large number of web servercomputers coupled to the database servers to serve content and makedatabase requests to update the database table when content is requestedby each client computer. The website may maintain one or more computersand/or processes to maintain coherency and/or execute the reaperprocesses and/or the aggregation processes.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 illustrates a network topology for determining the number ofclient computers currently connected to one or more server computers inreal-time in an example embodiment.

FIG. 2 illustrates a process for determining the number of clientcomputer connected to one or more server computers in real-time in anexample embodiment.

FIG. 3 is a block diagram of a plurality of sub-windows across multiplerolling windows in a server computer system with more than one servercomputer in an example embodiment.

FIG. 4 is a block diagram that illustrates a computer system upon whichan embodiment of the invention may be implemented.

While each of the drawing figures illustrates a particular embodimentfor purposes of illustrating a clear example, other embodiments mayomit, add to, reorder, and/or modify any of the elements shown in thedrawing figures. For purposes of illustrating clear examples, one ormore figures may be described with reference to one or more otherfigures, but using the particular arrangement illustrated in the one ormore other figures is not required in other embodiments.

TERMS, MEANINGS, AND EXAMPLES

For purposes of illustrating clear examples of the subject matterdiscussed in this disclosure, one or more meanings or examples may beassociated with one or more terms throughout this disclosure. However,each meaning or example associated with each term is not meant to beexclusive. Furthermore, words, such as “or” may be inclusive orexclusive unless expressly stated otherwise. The following is anon-exclusive list of meanings and examples of particular terms usedthroughout this disclosure, one or more terms in the following list maybe associate with one or more additional meanings or examples laterdiscussed herein.

A “computer” may be one or more physical computers, virtual computers,and/or computing devices. As an example, a computer may be one or moredesktop computers, laptop computers, mobile devices, cloud-basedcomputers, and cloud-based cluster of computers, virtual computerinstances or virtual computer elements such as virtual processors,storage and memory, and/or any other special-purpose computing devices.Any reference to “a computer” herein may mean one or more computers,unless expressly stated otherwise.

A “server computer” may be a computer that receives, processes, and/orresponds to one or more requests from one or more client computers.

A “client” or “client computer” may be a computer, or a process runningon a computer, that the requests data from, or sends data to, a servercomputer.

A “server computer system” may be a computer system comprising one ormore computers that receive, process, and respond to requests from oneor more client computers. For example, a website, cloud computingsystem, or content delivery network may be a server computer system.Computers in a server computer system may work in concert to receive,process, and respond to requests from a client computer.

A client is an “active client” or “actively coupled to” a servercomputer system while the client and server computer system arecommunicating with each other. For example, a client computer may beactively coupled to a server computer system while the client computersends one or more requests for data, receives the requested data, orsends one or more heartbeat messages. A client need not be persistentlyconnected to a server computer system for the client computer to beactively coupled to the server computer system. A client computer may beactively coupled to a server computer system while the client computeris intermittently requesting or receiving related data from a servercomputer system. Data may be related if the data is requested from thesame server computer system, or part of a whole. For example, if a movieis divided and stored in five separate files or if the five separatefiles are stored on the server computer system, then the five files arerelated. Accordingly, a client computer may be continuously and activelycoupled to the server computer system while the client computer sends aseparate request for each of the five files and downloads each of thefive files separately or concurrently.

A “session” is a period of time that a client computer is activelycoupled to a server computer system. For example, a session may beginand end when a continuous connection between a client computer and aserver computer system begins and ends, respectively. Also, for example,a session may begin when a client computer, or a web browser running ona client computer, sends a first request for a web page from a servercomputer system; throughout the session the client computer may send oneor more additional requests for one or more supporting assets referencedin the web page, such as images, media, JavaScript, stylesheets; and,the session may end when the client computer receives the web page andthe one or more supporting assets to present the web page to a user orwhen all connections between the client computer and the server computersystem over a particular protocol are terminated. Also for example, asession may begin when a client computer begins streaming, or requestinga portion of, a media title such as a movie, audiobook, or videopresentation; the session may end when the client computer finishesstreaming the media title, no longer requests any additional portion ofthe media title, or stops presenting the media title.

An “administrator” is a user that can receives, queries for, or modifiesthe state of a server computer system. The particular role of a user isnot critical and the label “administrator” is used here merely forconvenience to illustrate a clear example.

The notation <J, K> herein describes a list or vector of values. In theexample <J, K>, the vector has two values, and J represents the firstvalue in the vector and K represents the second value in the vector.Elements may be separated by spaces, commas, or any other character(s)to increase the ability to read and understand the description.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

General Overview

Techniques are described herein to allow a server computer system toestimate the number of clients actively coupled to the server computersystem in real-time, without locks, without client or user identifiers,with lower memory footprint/overhead (O(l)) memory) and a lower computerprocessor load than other techniques. Real-time may mean within aparticular window of time or soon after receiving data and/ordetermining results. Techniques are also discussed herein to calculateand reduce the expected error of the estimated number of active clients.

A server computer system may use the following equation to estimate thenumber of connections a server computer system has open (i.e., thenumber of clients that are actively coupled to the server computersystem):

$C \approx \frac{\sum\limits_{{t - w} < {m.{timestamp}} \leq t}\; {m.{delta}}}{w}$

In the above equation, C is the estimated number of connections as of acurrent time t, over a particular amount of time (“window”) w; thevalues w and delta in the equation have the same units of time, such asseconds or milliseconds. “m” represents a “heartbeat message” (discussedin further herein) that is associated with two values: a timestamp and adelta value. A server computer system may estimate the total number ofclient computers that are actively coupled to the server computer systemat a particular time t by summing delta values associated with heartbeatmessages that are associated with a timestamp within a particular amountof time w from the particular time t and dividing the sum by the amountof time in the window w. The sum of the delta values associated withheartbeat messages in a particular period of time, such as a window, maybe referred to herein as a total delta value.

In the above equation, a client identifier is not used. The same clientcomputer may send one or more heartbeat messages. However, the heartbeatmessages need not include a client identifier. Furthermore, a servercomputer system that receives the heartbeat messages need not attributeor associate the heartbeat messages with a particular client or user.

Although the above equation implies that heartbeat messages may bearchived or stored in persistent memory and later aggregated offline,one or more of the embodiments discussed herein illustrate how tocompute an estimated number of active client computers in real-timeusing data in volatile, non-persistent memory, without archivingheartbeat messages or waiting for an asynchronous, offline process to beexecuted using data or heartbeat messages archived or stored inpersistent memory. The estimated number of active client computerscoupled to a server computer system may be computed using a distributedcomputing system. Furthermore, the estimated number of active clientcomputers may be computed and emitted periodically throughout a widow,not just at the boundary of sequential contiguous windows.

Heartbeat Messages

A heartbeat message is a message a client computer sends to a servercomputer system that indicates, or is associated with, an amount of timesince the client computer sent the last heartbeat message to the servercomputer system during a particular session. The amount of timeindicated, or associated with, a heartbeat message is referred to as a“delta value”. The delta value may be represented in different ways. Forexample, a delta value may be a numerical value, such as an integer orfloating point value, which represents a particular number ofmilliseconds, seconds, minutes, or any other unit of time, since theclient computer last sent a heartbeat message to the server computersystem during a particular session. If the client computer is beginninga session with a server computer system (i.e., just starting to beactively coupled to the server computer system), then the first deltavalue in the first heartbeat message of the session may be zero. If thedelta value is 500, then the delta value may indicate that 500milliseconds have passed since the client computer sent the previousheartbeat in the current session or the session started 500 millisecondsago.

In an embodiment, a heartbeat message may include a value from which thedelta value may be derived. For example, a heartbeat message may includea timestamp that corresponds to a time at which the last heartbeatmessage was sent by the client computer to the server computer systemduring a current session or the beginning of the current session. Theserver computer system may determine the delta value for the heartbeatmessage by determining the amount of time that has elapsed from the timethat the timestamp corresponds to, and the current time.

In an embodiment, heartbeat messages are sent at a particular frequencyor predefined interval. Accordingly, the heartbeat message need notinclude a delta value, timestamp, or any other payload. The servercomputer system may assume that the delta value is the span of thepredefined interval. The server computer system may modify thepredefined interval by sending data to one or more client computers thatinstructs the client computers to send heartbeat messages at a modifiedfrequency or interval. Accordingly, the server computer system mayassume that the delta value for each heartbeat is the modified interval.

A heartbeat message may comprise data included in one or more datapackets sent electronically using one or more network protocols, such asUser Datagram Protocol, Transmission Control Protocol/Internet Protocol,HyperText Transfer Protocol, or any other standard or proprietarycomputer networking protocol. A heartbeat message may also include on ormore attributes discussed further herein. A heartbeat message mayinclude the delta value or one or more attributes in binary, plain text,structured data format (such as HTML or JSON), or any other standard orproprietary format or encoding.

A server computer system may associate, or add data to, a heartbeatmessage that corresponds to a time that the heartbeat message wasreceived from a client computer. For example, when a server computersystem receives a heartbeat message from a client computer, the servercomputer system may associate a timestamp to the heartbeat message,wherein the timestamp indicates, or corresponds to, the time that theheartbeat message was received or processed by the server computersystem. A server computer can use the timestamp to determine theheartbeat message belongs to a particular window or sub-window(discussed further herein). In an embodiment, a client computer mayinclude a timestamp in the heartbeat message, wherein the timestampindicates, or corresponds to, the time the client computer sent theheartbeat message.

Heartbeat messages may include additional data to specify one or moreclassifications or groupings. For example, a heartbeat message mayinclude data indicating one or more attributes, such as the geographicregion the client resides, the type of client computer sent theheartbeat, the service the client is using, the data or type of data theclient is requesting. For example, a heartbeat message may includeattributes such as “web page”, “image”, “audio”, “video”, or the name ofa particular webpage, image, audio book, movie, asset, or dataset.

Example Network Topology

FIG. 1 illustrates a network topology for determining the number ofclient computers currently connected to one or more server computers inreal-time in an example embodiment. In FIG. 1, system 100 comprisesserver computer system 110, client computer 180 and client computer 190communicatively coupled over one or more computer networks.

Client Computers

Client computer 180 and client computer 190 are client computersoperably coupled to server computer system 110. For purposes ofillustrating a clear example, only two client computers, each of whichare illustrated as having a single connection to server computer system110, are included in system 100; however, any number of client computersmay be actively coupled to server computer system 110 intermittently orat the same time. Furthermore, a client computer, such as clientcomputer 180, may maintain more than one concurrent active coupling toserver computer system 110.

A client computer may determine when a session begins and ends based onwhat the client is configured to do. For purposes of illustrating aclear example, assume a client application running on a client computeris a video player that downloads and presents media titles, such as amovies or audiobooks, from a server computer system. The client maydetermine that a session begins when the client begins downloading,streaming, or requesting at least a portion of the media title selectedby a user. The client computer may determine that the session ends whenthe client computer finishes streaming or downloading the media titlefrom the server computer system, no longer requests any additionalportion of the media title from the server computer system, or stopspresenting the media title to a user via a display or speakers.

A client may determine a delta value, i.e., how much time has passedsince the beginning of a current session or how much time has passedsince a previously sent heartbeat. In an embodiment, a client maydetermine a delta value by maintaining a single value, such as anunsigned or 64-bit integer. For example, a client computer may maintaina starting timestamp value represented as an unsigned 64-bit integerthat indicates the number of milliseconds since epoch that the clientstarted a current session or sent a previous heartbeat message in thecurrent session. The client computer may determine how much time haspassed since the beginning of the current session by generating acurrent timestamp value represented as another unsigned 64-bit integerthat indicates the number of milliseconds since epoch, and subtractingthe starting timestamp value from the current timestamp value. Theclient may replace the starting timestamp with the current timestampvalue.

A client can send heartbeat messages to a server computer system atrandom times during a session, at regular intervals during a session, oras instructed by the server computer system. For example, a clientcomputer may have a callback method that is called periodicallyaccording to a frequency stored locally or received from a clientcomputer system. Each time the callback method is called, the clientcomputer determines a delta value, sets the starting timestamp to be thecurrent timestamp, and sends the delta value to the server computersystem. Also for example, a client computer may have a heartbeat methodthat is called each time data is requested from a server computersystem. Each time the heartbeat method is called, the client computerdetermines a delta value, updates the starting timestamp to be thecurrent timestamp, and sends a heartbeat message with the delta value tothe server computer system.

The server computer system may send data to the client computer thatindicates when the client computer should send future heartbeats. Forexample, a server computer system may send data to a client computerthat indicates the frequency at which the client should send heartbeatmessages.

A heartbeat protocol defines when a session begins and ends or when to aclient should send heartbeat messages. As discussed above, a client or aserver computer system may define the parameters of a heartbeatprotocol, i.e., when a session begins and ends or when a client shouldsend heartbeat messages. A client computer, such as client computer 180,may execute two or more client processes concurrently, such as a webbrowser and a video player, each of which may be downloading web pagesor video from, and actively coupled to, server computer system 110 oranother server computer system concurrently. Each of the clientprocesses may send heartbeat messages to the same or different servercomputer systems according to different heartbeat protocols.Accordingly, each heartbeat protocol may define when a session beginsand ends differently than, or independently from, another heartbeatprotocol; and, each heartbeat protocol may define when heartbeatmessages are sent differently than, or independently from, anotherheartbeat protocol.

Server Computer System

Server computer system 110 comprises three server computers and astorage system operably coupled together: server computer 120, servercomputer 130, server computer 140, and storage 150. In otherembodiments, server computer system 110 may include one or morecomputers distributed over one or more computer network or geographicregions.

Server computer system 110 estimates how many clients are activelycoupled to server computer system 110. Server computer system 110 mayalso compute the expected error of the estimation, discussed furtherherein. Server computer system 110 may send new heartbeat protocolparameters to active clients indicating that clients should sendheartbeat messages more or less frequently, which reduces or increasesthe expected error, respectively, as discussed further herein. Servercomputer system 110 may send new heartbeat protocol parameters to activeclients in response to input from a network administrator or variousload metrics of server computer system 110, such as available bandwidth,observed latency, or any other computer-based load metric.

Server computer system determines when a window or sub-window (discussedfurther herein) begins and ends or the amount of time each window orsub-window spans. For example, an administrator may store data instorage 150 that indicates a window begins and ends every second, andeach window spans a single second. Accordingly, for each second, servercomputer system 110 sets a total delta value to zero; for each heartbeatmessage received during the one-second window, the server computersystem 110 adds the delta value associated with the heartbeat message tothe total delta value; and at the end of the window, the server computersystem 110 estimates the number of clients actively coupled to theserver computer system based on the total delta value and emits theestimate.

Storage 150 may be volatile or persistent storage. Clients, such asclient computer 180 and client computer 190, may request data fromstorage 150, such as a particular frequency over which clients shouldsend heartbeat messages. Storage 150 may store one or moreconfigurations that indicate the parameters over which server computersystem 110 or clients should operate. For example, storage 150 mayinclude data that indicates the following:

-   -   A window should be 3000 milliseconds;    -   The estimated number of active clients should be bound to a        minimum expected error threshold;    -   The estimated number of active clients should be bound to a        maximum expected error threshold;    -   Server computer 120 is the master server computer (a computer        configured to combine the data from other server computers to        estimate the total number of client actively coupled to a server        computer system);    -   Where results of each window should be stored on storage 150;        and    -   Whether a result is sent to a remote client computer operated by        an administrator of server computer system 110.

Storage 150 may be shared memory or distributed memory that is operablycoupled to each of the one or more computers in server computer system110. Storage 150 may be included in a computer, such as server computer120, or a different computer not illustrated in system 100.

While some of the components listed above are illustrated as if runningon a separate or remote computer from each another computer, at leastsome of the components listed above may be part of, or executed on, thesame computer. For example, server computer 120, server computer 130server computer 140 and storage 150 may be components or processesexecuted on the same single, physical computer. Also for example, servercomputer 120 and storage 150 may be part of the same computer and may beoperably coupled to other server computers in server computer system110, such as server computer 130 and server computer 140.

Example Process for Estimating the Number of Clients Actively Coupled toa Server Computer System

FIG. 2 illustrates a process for estimating the number of clientcomputers actively coupled to a server computer system in real-time inan example embodiment. The steps in FIG. 2 are not based on a clientidentifier; accordingly a heartbeat message need not include a clientidentifier. Although, the same client computer may send one or moreheartbeat messages, the heartbeat messages need not include a client oruser identifier. Furthermore, a server computer system that receives theheartbeat messages need not attribute or associate the heartbeatmessages with a particular client or user. In an embodiment, a heartbeatmessage can include a client or user identifier as an attribute or forpurposes of classification discussed further herein.

In step 210, a server computer system receives a plurality of heartbeatmessages wherein each heartbeat message indicates a delta value. Forexample, client computer 180, client computer 190, and one or more otherclients may each send a heartbeat message to server computer 120. Eachheartbeat message is associated with a delta value. For example, eachheartbeat message may include or imply a delta value as discussedherein.

In step 220, server computer 120 determines a total amount of timebased, at least in part, on the delta value from each heartbeat message.There are many ways to determine the total amount of time. For purposesof illustrating a clear example, assume server computer 120 receives theplurality of heartbeat messages associated with a particular window.Server computer 120 sets a total delta value associated with theparticular window to zero; and, for each heartbeat message, servercomputer 120 adds the delta value associated with the heartbeat messageto the total delta value.

While step 210 and 220 are illustrated as if sequential steps, thesesteps may be executed, at least in part, concurrently. For example, atthe beginning of a window, server computer 120 may set a total deltavalue to zero. As server computer 120 receives or processes heartbeatmessages from client computer 180, client computer 190, or one or moreother clients during the window, server computer 120 adds the deltavalue associated with each heartbeat message to the total delta value.For purposes of illustrating a clear example, assume server computer 120received 10,000 heartbeat messages, each of which is associated with afour-second delta value. As server computer receives or processes eachheartbeat message of the 10,000 heartbeat messages, server computer 120adds four seconds to a total delta value. Accordingly, after servercomputer 120 processes the last heartbeat of the 10,000 heartbeatmessages, server computer 120 has calculated the total delta value to be40,000 seconds.

In step 230, the server computer system determines a total number ofactive client computers based, at least in part on the total amount oftime and the defined amount of time in a window. For purposes ofillustrating a clear example, assume a window spans two seconds, and, atthe end of the window, the total delta value associated with the windowis 40,000. Server computer 120 divides 40,000 by 2, which generates anestimate of 20,000 clients actively coupled to server computer system110.

Emitting an Estimated Number of Active Clients

In step 240, the server computer system emits the total number of activeclient computers coupled to the server computer system. Emitting anestimate means electronically sending the estimate to another servercomputer, caching the estimate, storing the estimate in a persistentdata store, or responding to the estimate with one or more actions. Forexample, server computer 120 may send the estimated number of activeclients to a computer, such as a phone or desktop computer, used by theadministrator of server computer system 110. Additionally oralternatively, server computer 120 may store the estimated number ofactive clients coupled to server computer system 110 for a particularwindow along with data that identifies, or associates the estimatednumber with, the particular window in storage 150. An administrator mayuse a computer, such as client computer 190 or another computer notillustrated in FIG. 1, but operably coupled to server computer system110, to query for the estimated number of clients actively coupled toserver computer system 110 over a period of time that includes at leastthe particular window. In response, storage 150 may send the estimatenumber of active clients for the particular window to theadministrator's computer for visualization and/or other user.

Modifying the Capacity or Behavior of the Server Computer System or Oneor More Clients

A server computer system may perform one or more actions that modify thecapacity or behavior of the server computer system or one or moreclients based on, or in response to, one or more estimated numbers ofactive client computers. For purposes of illustrating a clear example,assume the following:

-   -   Server computer system 110 is part of an elastic cloud computing        system;    -   Server computer system 110 is configured to support up to 1,000        active clients streaming video content;    -   A maximum active client threshold is set to 1,000;    -   A minimum active client threshold is set to 800;    -   A duration threshold is less than or equal to the amount of time        from the beginning of the first window and the end of the third        window;    -   The first most recent emitted estimate of active clients is        1,010;    -   The second most recent emitted estimate of active clients is        1,022; and    -   The third most recent emitted estimate of active clients is        1,052.

Server computer 120 compares the maximum active client threshold and theminimum active client threshold to the first most recent emittedestimate of active clients, the second most recent emitted estimate ofactive clients, and the third most recent emitted estimate of activeclients.

In response to determining that the estimates satisfy the maximum activeclient threshold for the duration threshold (e.g., each of theestimates, an average of the estimates, the median of the estimates, orsome other metric based on the estimates, is equal to or greater thanthe maximum active client threshold, and the estimates correspond towindows that at least collectively span a period of time that is equalto or greater than the duration threshold), server computer 120allocates additional computing resources, such as additional servercomputers, to server computer system 110 through the elastic cloudcomputing system's API to support additional active client computers.Based on the new total number of computing resources allocated to servercomputer system 110, server computer 120 may determine a new maximumactive client threshold and update the maximum active client threshold.If, in the current example, a new server computer is estimated tosupport up to 300 new active clients streaming video, then after the newserver computer is allocated to server computer system 110, servercomputer 120 updates the maximum active client threshold to 1,300 andthe minimum active client threshold to 1,100.

If server computer 120 determines the estimates satisfy the minimumactive client threshold for a duration threshold (e.g., each of theestimates, an average of the estimates, the median of the estimates, orsome other metric based on the estimates, is equal to or below theminimum active client threshold for a period of time that is equal to orgreater than the duration threshold), then server computer 120deallocates one or more computing resources, such as server computer 130or server computer 140, from server computer system 110. In response todeallocating resources to server computer system 110, server computer120 lowers the minimum active client threshold or maximum active clientthreshold accordingly.

A server computer system may take one or more other actions. Forexample, if an emitted estimate is greater than a maximum active clientthreshold, then the server computer system pushes an alert to a deviceoperated by an administrator that includes the estimated number ofactive clients or indicates that the estimate is over the threshold.Additionally or alternatively, in response to emitting an active clientestimate, server computer system 110 cause one or more clients to changetheir behavior. For example, and as discussed further herein, servercomputer system 110 may adjust the frequency of heartbeat messages sentby one or more clients based on one or more estimated number ofestimated number of active clients over one or more windows.

Adjusting Heartbeat Frequency Based on Server Computer System Resourcesor Expected Error

Returning to FIG. 2, in step 250, the server computer system determinesa frequency at which one or more client computers should send heartbeatmessages. The frequency at which a client computer sends heartbeatmessages is referred to herein as a heartbeat frequency. For purposes ofillustrating clear examples herein, a heartbeat frequency is discussedas a rate that indicates how often a client should send a heartbeatmessage for a given period of time, such as once message per second, or120 times per minute. Accordingly, a client will send more heartbeatmessages over a particular amount of time when the heartbeat frequencyis increased and fewer heartbeat messages over the same amount of timewhen the heartbeat frequency is decreased. However, in an embodiment, aheartbeat frequency may indicate a how much time should pass betweensending heartbeat messages, such as 100 milliseconds or two seconds.Accordingly, a client will send more heartbeat messages over aparticular amount of time when the heartbeat frequency is decreased andfewer heartbeat messages over the same amount of time when the heartbeatfrequency is increased.

The server computer system may autonomously cause one or more clientcomputers to change their heartbeat frequency based on one or morefactors, such as available computing power, available bandwidth,expected error (discussed further herein), or one or more other factorsdiscussed herein. For purposes of illustrating a clear example, assume aheartbeat frequency value indicates the frequency at which clientsshould sent heartbeat messages to server computer system 110 and servercomputer 120 monitors the available bandwidth of server computer system110. If server computer 120 determines the available bandwidth is belowa threshold for a particular duration, then server computer 120 adjuststhe heartbeat frequency value stored in storage 150 to decrease thefrequency at which clients send heartbeat messages.

The heartbeat frequency may change based on input from an administrator.For example, if the load on one or more computers in server computersystem 110 is above a particular threshold, then an administrator mayadjust a heartbeat frequency value stored in storage 150 to increase thefrequency at which clients should send heartbeats. An administrator mayadjust the frequency for one or more other reasons, such as availablebandwidth or expected error.

In step 260, the server computer system sends the frequency to one ormore client computers. For example, in response to receiving a heartbeatmessage from client computer 190, server computer 120 may send theheartbeat frequency value stored on storage 150 to client computer 190.Accordingly, clients that receive the new heartbeat frequency value willbegin sending heartbeat messages according to the heartbeat frequencyvalue. The heartbeat frequency value may be sent to a client in responseto a heartbeat message or one or more other requests for data.Additionally or alternatively, a client may send server computer system110 a request for a heartbeat frequency, and in response, servercomputer system 110 sends the heartbeat frequency value to the clientcomputer.

Distributed Server Computer System

Heartbeat messages may be received and processed by a plurality ofserver computers in a server computer system. For example, servercomputer 120, server computer 130, and server computer 140 may eachreceive a plurality of heartbeat messages from clients over a particularwindow of time. Server computer 120 computes a first partial total deltavalue, server computer 130 computes a second partial total delta value,and server computer 140 computes a third partial total delta value.Server computer 120, server computer 130, and server computer 140 maysend the partial total delta values to a master computer defined by datain storage 150. For purposes of illustrating a clear example, assumeserver computer 120 is the master computer. Server computer 120 computesa combined total delta value for the window by summing each of thepartial delta values received for the window. Server computer 120estimates the number of clients actively coupled to server computersystem 110 by dividing the combined total delta value by the span of thewindow.

A server computer system may estimate the number of clients activelycoupled to the server computer system without regard to which servercomputer received or processed each heartbeat message. Accordingly,clients may choose which server computer to send each heartbeat messageusing a local choice algorithm or a load balancer. For example, inresponse to sending a first heartbeat message, server computer 120 maysend, and client computer 180 may receive, a list of available servercomputers to send heartbeat messages to in server computer system 110.Client computer 180 may send heartbeat messages to each of the servercomputers in the list according to a uniform distribution or round-robinmethod. Additionally or alternatively, a server-side load balancer maydetermine which server computer receives or processes each heartbeatmessage.

Scaling

In an embodiment, clients may send too many heartbeat messages for asingle server computer to receive and process quickly. Whether aheartbeat message is processed quickly can be based on one or moremetrics. In an embodiment, a heartbeat message is processed quickly ifthe server computer that receives the heartbeat message and adds thedelta value associated with the heartbeat message to a total deltavalue, partial total delta value, sub-total delta value (discussedfurther herein), or attributed total delta value (discussed furtherherein) within the same window.

Server computer system 110 may determine whether heartbeat messages arebeing processed quickly. For example, storage 150 may include a minimumprocessing threshold that indicates the percentage of heartbeats thatshould be processed quickly in each window for a duration threshold.Server computer system 110 can determine what percentage of heartbeatmessages are being processed quickly by server computer system 110. Ifthe percentage is less than the minimum process threshold for an amountof time greater than the duration threshold, then an administrator mayapply additional computational resources, such as additional servercomputers, to server computer system 110. Additionally or alternatively,server computer system 110 may be configured to autonomously addadditional server computers. For example, if server computer system 110is part of an elastic computer system, then server computer 120 mayrequest additional server computers from the elastic computer systemthrough an API. Additionally or alternatively, the administrator maydecrease the frequency at which clients should send heartbeat messages.If a server computer is not processing heartbeat messages quickly, thenthe server computer system may be referred to as overloaded.

Server computer system 110 may determine whether excess computationalresources are allocated to processing heartbeats. For purposes ofillustrating a clear example, assume storage 150 includes a satisfactoryprocessing threshold that indicates the percentage of heartbeats thatneed not be processed quickly in each window for a duration threshold.Server computer 120 determines what percentage of heartbeat messages arebeing processed quickly by server computer system 110. If the percentageis greater than the satisfactory processing threshold for an amount oftime greater than the duration threshold, then an administrator maydeallocate computational resources, such as one or more servercomputers, to server computer system 110. Additionally or alternatively,server computer system 110 may be configured to autonomously deallocatecomputational resources. For example, if server computer system 110 ispart of an elastic computer system, then server computer 120 may requestthat server computer 130 be deallocated or released from server computersystem 110 into the elastic computer system through an API. Additionallyor alternatively, the administrator may increase the frequency at whichclients should send heartbeat messages.

Expected Error

A server computer system can calculate an expected error. For example,assume the following:

-   -   The time at which each client sends heartbeat messages with        respect to a window boundary is uniformly distributed, i.e., for        each heartbeat message at time t₀ if the current window started        at time t_(w), then t₀−t_(w) is uniformly distributed between        zero and the length of the window w. This is true for        embodiments where there are no events causing multiple clients        to send heartbeats at the same time. For example, assume users        using a plurality of video playing devices independently select        one or more videos to download and watch at different times of        during the day from a video streaming service. If each client        begins sending heartbeat messages according to a heartbeat        frequency when the user using the client selects a video to        stream from a server computer system, then the times at which        each client sends a heartbeat message, and the times the server        computer system receives the heartbeat messages from the        clients, will be uniformly distributed.    -   The number of active clients is large with respect to the Law of        Large Numbers.    -   The same clients are active throughout an entire window.

Under the forgoing, the expected value, or mean, of C, denoted as E[C],is equal to n, where n is the total number of active clients. Thestandard deviation of C, denoted as σ(C), can be computed using thefollowing equation:

${\sigma (C)} = {\frac{T}{2\; w\sqrt{n}}.}$

In the equation above, w is the size of a window in seconds; Tis thequadratic mean of the frequency at which each client is sendingheartbeat messages, i.e., the square root of the mean of the squaredlength of the heartbeat frequency.

By the law of large numbers, the C is normally distributed with theabove mean, E(C), and standard deviation, σ(C). Accordingly, C, theestimate of the number of clients is unbiased. As clients increase thefrequency at which heartbeat messages are sent (i.e., reduce the amountof time between sending heartbeat messages), the error in the estimate Cis reduced. Furthermore, as the server computer system increases thelength of w, the error in the estimate C is reduced. If the number ofclients actively coupled to a server computer system changes during awindow, then the expected value of the estimate C is the average numberof clients during the window, and the standard deviation σ(C) is boundedby the standard deviation at the time within the window at which thefewest clients are actively coupled to the server computer system.

Given a fixed T for near real-time estimates on the number of clientsactively coupled to the server computer system, an administrator for theserver computer system may pick a w small enough so that few clientsbecome active or inactive within the same window, but large compared toT to reduce the error. In an embodiment, an administrator of the servercomputer system may select a value of w close to the geometric mean of Tand the expected length of each client's active state.

Modify Capacity or Behavior Based on Expected Error

Using the equations discussed above, an administrator may select aheartbeat frequency and a window length; in response, the servercomputer system may determine the expected error based on the heartbeatfrequency and the window length. Accordingly, an administrator mayselect a target expected error value and a window length; in response,the server computer system may determine the required heartbeatfrequency at which the expected error matches the target expected errorgiven the window length set by the administrator. Additionally oralternatively, an administrator may select a target expected error valueand a heartbeat frequency; in response, the server computer system maydetermine the required window length at which the expected error matchesthe target expected error given the heartbeat frequency set by theadministrator.

A server computer system may modify the heartbeat frequency or thelength of a window based on a target expected error. For purposes ofillustrating a clear example, assume server computer 120 determines thatserver computer system 110 is overloaded. In response, server computer120 may decrease the heartbeat frequency as discussed herein, whichreduces the load on server computer system 110. Server computer 120determines a new window length based on the new heartbeat frequency tomaintain the target expected error. In an embodiment, the heartbeatfrequency is reduced and the window length is recalculated in responseto determining that additional computer resources are not available dueto cost, availability, or one or more other reasons.

In an embodiment, the window length may be adjusted in response to inputfrom an administrator. If the window length is adjusted, then the servercomputer system may adjust the heartbeat frequency accordingly. Forexample, in response to receiving input from an administrator to reducethe window length to a new window length, server computer 120 calculatesa new heartbeat frequency based on the target expected error and the newwindow length. However, increasing the heartbeat frequency increases theload on the server computer system. Accordingly, in response toadjusting the heartbeat frequency, a server computer may determinewhether computer resources should be allocated or deallocated. Forexample, if server computer 120 determines that the load on the servercomputer system from the new heartbeat frequency would overload servercomputer system 110, or that server computer system 110 is alreadyoverloaded, then server computer 120 may allocate one or more computerresources to server computer system 110 accordingly. In response toreducing a heartbeat frequency or determining that server computersystem 110 is not overloaded, server computer 120 may deallocate one ormore computing resources from server computer system 110 accordingly.

Sub-Windows and Rolling Windows

Using one or more of the methods discussed above, a server computersystem may estimate and emit the number of clients connected to theserver computer system at the end of every window. For real-timeestimates shorter than the span of a window, a server computer systemmay use sub-windows and rolling windows.

A sub-window is shorter than a single window. Each sub-window in aseries of sub-windows spans an equal amount of time. Each sub-window isassociated with a subtotal delta value. To compute the total delta valuefor a window, a server computer system adds the subtotal delta valuesassociated with each sub-window within the span of the window.

A rolling window is a period of time that spans the length of a fullwindow; however, a first rolling window can end after a second windowbegins. A server computer system can estimate and emit the number ofclients actively coupled to the server computer system at the end ofeach rolling window. Accordingly, a server computer system can emit anestimate in real-time: more than one time over the duration or span of asingle window.

FIG. 3 is a block diagram of a plurality of sub-windows across multiplerolling windows in a server computer system with more than one servercomputer in an example embodiment. In FIG. 3, window 310 and window 320are rolling windows that each span one or more of the same sub-windows.For purposes of illustrating a clear example, assume the following:

-   -   Sub-window 332, sub-window 334, sub-window 336, and sub-window        338 are each sub-windows associated with a subtotal generated by        server computer 120 from the delta values associated with the        heartbeat messages received or processed by server computer 120        during the sub-window.    -   Sub-window 342, sub-window 344, sub-window 346, and sub-window        348 are each sub-windows associated with a subtotal generated by        server computer 130 from the delta values associated with the        heartbeat messages received or processed by server computer 130        during the sub-window;    -   Server computer 120 is the master computer for estimating the        number of clients actively coupled to server computer system 110        during each rolling window, and accordingly, server computer 130        sends each of the sub-window produced by server computer 130 to        server computer 120.

Server computer 120 estimates and emits the total number of clientscoupled to server computer system 110 for rolling window 310 by summingthe subtotal delta values associated with each of the followingsub-windows:

-   -   Sub-window 332, sub-window 334, and sub-window 336, which were        generated by server computer 120; and    -   Sub-window 342, sub-window 344, and sub-window 346, which were        each generated by server computer 130 and sent to server        computer 120.

Server computer 120 estimates and emits the total number of clientscoupled to server computer system 110 for rolling window 320 by summingthe subtotal delta values associated with each of the followingsub-windows:

-   -   Sub-window 334, sub-window 336, and sub-window 338, which were        generated by server computer 120;    -   Sub-window 344, sub-window 346, and sub-window 348, which were        each generated by server computer 130 and sent to server        computer 120.

Accordingly, server computer system 110 can emit a new estimate of thenumber of active clients at the end of each sub-window. Server computersystem 110 may, but need not, retain the sub-windows in memory that areolder than a full window. For example, after sub-window 332 andsub-window 342 are no longer within the span of a rolling sub-window, ofwhich an estimated total number of active clients has been emitted,server computer 120 may delete, deallocate, or overwrite sub-window 332and sub-window 342 in memory.

Keyed Counts

As discussed herein, a heartbeat message may include additional data toclassify the heartbeat message into one or more groups. For example, aheartbeat message may include two attributes: a first attribute thatidentifies a geographic region from which the client is sending theheartbeat message, and a second attribute that indicates the type ofclient computer that is sending the heartbeat message.

A server computer system may generate a total delta value or anestimated number of active clients for each classification or attributefor each window. The total delta values may be stored in a hash tableassociated with each window. For example, for each heartbeat message ofa plurality of heartbeat messages, when server computer 120 receives orprocesses the heartbeat message, for each attribute in the heartbeatmessage: server computer 120 determines whether an entry in the hashtable includes a hash key that corresponds to the attribute; if no suchentry is in the hash table, then server computer 120 generates a newentry in the hash table that includes the hash key and a total deltavalue that is equal to the delta value in the message; otherwise, servercomputer 120 adds the delta value in the message to the total deltavalue for the entry that includes the hash key. After the window closes,server computer 120 estimates the number of active clients for eachattribute. For example, for each entry in the hash table, servercomputer 120 divides the total delta value for the entry by the span ofthe window, and stores the estimated number of active clients in theentry.

For purposes of illustrating a clear example, assume the following:

-   -   Server computer 120 receives or processes a first heartbeat        message in a first window with a delta value set to one and the        following attribute labels: “mobile device” and “CA”;    -   Server computer 120 receives or processes a second heartbeat        message in the first window with a delta value set to one and        the following attribute labels: “mobile device” and “NY”;    -   Server computer 120 maintains a hash table associated with the        first window.

When server computer 120 processes the first heartbeat message, servercomputer 120 generates a first entry and a second entry in the hashtable associated with the first window that correspond to the twoattributes included in the first heartbeat message: “mobile device” and“CA”, respectively. Server computer 120 sets the total delta value foreach entry to the delta value in the first heartbeat message, which inthis example is one. Accordingly, the entries in the hash table afterserver computer 120 processes the first heartbeat are <“mobile device”,1> and <“CA”, 1>, wherein, for each entry, the first value is the hashkey and the second value is the total delta value. Additionally, servercomputer 120 may add the delta value from the first heartbeat message toa total delta value associated with the first window. Accordingly, afterthe first heartbeat message is processed, the total delta valueassociated with the first window is one.

When server computer 120 processes the second heartbeat message, servercomputer 120 determines that the first entry has a hash key thatcorresponds to “mobile device”, but not “NY”. In response, servercomputer 120 adds the delta value for the second heartbeat message tothe total delta value associated with the first entry, and generates athird entry in the hash table that corresponds to the attribute “NY” andsets the total delta value that corresponds to the third entry in thehash table to one. Accordingly, the entries in the hash table afterserver computer 120 processes the second heartbeat message are <“mobiledevice”, 2>, <“CA”, 1>, and <“NY”, 1>. Additionally, server computer 120may add the delta value from the second heartbeat message to the totaldelta value associated with the first window. Accordingly, after thesecond heartbeat message is processed, the total delta value associatedwith the first window is two.

Server computer system 110 may continue to update the hash table foreach heartbeat message received during the first window. At the end ofthe first window, server computer 120 generates the estimated number ofactive clients for each entry in the hash table. For example, for eachentry in the hash table, server computer 120 divides the total deltaassociated with the entry by the amount of time the first window spans,and stores the result in the entry. Assume that in this example the spanof the first window is two seconds. Accordingly, after server computer120 generates the estimate number of active clients for each entry, theentries in the hash table are <“mobile device”, 2, 1>, <“CA”, 1, 0.5>,and <“NY”, 1, 0.5>, wherein, for each entry, the first value is the hashkey, the second value is the total delta time, and the third value isthe estimated number of active clients. Additionally or alternatively,server computer 120 generates a new hash table to store the number ofactive clients associated with each hash key. Accordingly, for eachentry in the previous hash table, server computer 120 generates a newentry in the new hash table that includes the same hash key and theestimated number of active clients. Additionally, server computer 120divides the total delta value associated with the first window by thespan of the first window. Accordingly, the estimated number of activeclients associated with the first window is one.

Server computer 120 may emit the window with a total delta value, thehash table for the window, or data from the hash table for the window asdiscussed above. For example, server computer 120 may store the hashtable or the estimated number of active clients for eachclassification/attribute in storage 150. An administrator, using aclient computer, may query storage 150 for the hash tables associatedwith each window over a particular period of time to generate one ormore reports about where clients are from or what types of devicesclients are using.

In an embodiment, the total delta value or total number of activeclients associated with each entry in a hash table may be a 64-bitunsigned integer. Using a hash table or 64-bit unsigned integers is onea way to implement the features discussed above. Additionally oralternatively, one or more other types of data or data structures may beused.

Lazy Instantiation of Hash Tables and Entries

Server computer system 110 may generate a new hash table at thebeginning of each window, or a server computer system may lazilygenerate hash tables and entries. For purposes of illustrating a clearexample, assume the following:

-   -   Server computer 120 receives or processes a third heartbeat        message during a second window that does not include any        attributes;    -   Subsequently, server computer 120 receives a fourth heartbeat        message during the second window that does include one or more        attributes.

In response to receiving or processing the third heartbeat message,server computer 120 may add the delta value associated with the thirdheartbeat message to a total delta value associated with the secondwindow. Server computer 120 need not generate a hash table for thesecond window or any entries that correspond to anyclassifications/attributes. In response to receiving the fourthheartbeat message, which includes one or more attributes, servercomputer 120 determines that no hash table is associated with the secondwindow yet, and generates the hash table, and one or more entries thatcorrespond with the one or more attributes in the fourth heartbeatmessage as discussed herein.

Keyed Counts Over Rolling Windows and Distributed Systems

The features discussed in this section may be used for rolling windowsand sub-windows. For example, a subtotal hash table may be created for,and associated with, each sub-window. When server computer system 110generates or emits a rolling window, then server computer system 110 maygenerate an aggregated hash table. Each entry in the aggregated hashtable corresponds to one or more entries in one or more subtotal hashtables that are associated with one or more sub-windows in the rollingwindow. Each aggregated total delta value associated with an entry inthe aggregated hash table is the sum of the one or more subtotal deltavalues associated with the one or more corresponding entries in the oneor more subtotal hash tables associated with the one or more sub-windowsin the rolling window. For each entry in the aggregated hash table,server computer system 110 computes an aggregated estimated number ofactive clients for the entry by dividing the aggregated total deltavalue associated with the entry by the span of the rolling window. Theaggregated estimated number of active clients may be associated with theentries in the aggregated hash table or a new hash table as discussedherein.

The features discussed in this section may be used for partial windows.For example, a partial hash table may be created for, and associatedwith, each partial window generated on a server computer in servercomputer system 110. Server computer system 110 may generate anaggregated hash table. Each entry in the aggregated hash tablecorresponds to one or more entries in one or more subtotal hash tablesthat are associated with one or more partial windows in the window. Eachaggregated total delta value associated with an entry in the aggregatedhash table is the sum of the one or more partial total delta valuesassociated with the one or more corresponding entries in the one or moresubtotal hash tables associated with the one or more partial windows inthe window. For each entry in the aggregated hash table, server computersystem 110 computes an aggregated estimated number of active clients forthe entry by dividing the aggregated total delta value associated withthe entry by the span of the rolling window. The aggregated estimatednumbers of active clients may be associated with the entries in theaggregated hash table or a new hash table as discussed herein.

For purposes of illustrating a clear example of using one or more of thefeatures discuss herein for partial windows and sub-windows, assume thefollowing:

-   -   Server computer 120 generates a first partial window with a        first subtotal hash table; the first subtotal hash table has two        entries: <“mobile computer”, 2.2> and <“NYC”, 2>.    -   Server computer 130 generates a second partial window with a        second subtotal hash table; the second partial window        corresponds to the same time as the first partial window; the        second subtotal hash table has one entry: <“mobile computer”,        2>.    -   Server computer 120 generates a third partial window with a        third subtotal hash table; the third subtotal hash table has one        entry: <“desktop computer”, 1>.    -   Server computer 130 generates a fourth partial window with a        fourth subtotal hash table; the fourth partial window        corresponds to the same time as the third partial window; the        fourth subtotal hash table has one entry: <“mobile computer”,        2>.    -   The span of a rolling window is two seconds opening at a        particular beginning time and closing a particular ending time.    -   The first partial window, the second partial window, the third        partial window, and the fourth partial window are within the        span of the rolling window.    -   The span of each sub-window is one second, and the unit of time        for each delta time, total delta time, or subtotal delta time is        in seconds.    -   Server computer 120 is the master server computer and receives        the second partial window and the fourth partial window from        server computer 130 after server computer 130 determines that        the second partial window has closed and the fourth partial        window has closed, respectively.

Server computer 120 generates an aggregate hash table for the rollingwindow based on the subtotal hash tables from each of the four windows.Specifically, server computer 120 generates three entries in theaggregate hash table: <“mobile computer”, 6.2>, <“NYC”, 2>, and<“desktop computer”, 1>. For each entry in the aggregate hash table,server computer 120 divides the aggregate total delta value by theamount of time that the rolling window spans to produce an aggregateestimated number of active clients. In this example, server computer 120generates a new hash table with entries for each of the aggregateestimated number of active clients, referred to as an aggregateestimated number of active clients hash table, which includes thefollowing entries: <“mobile computer”, 3.1>, <“NYC”, 1>, and <“desktopcomputer”, 0.5>, wherein, for each entry, the first value is the hashkey for a corresponding classification/attribute and the second value isthe aggregate estimated number of clients for the correspondingclassification/attribute. As discussed herein, server computer 120 mayemit the rolling window, the aggregate estimated number of activeclients hash table, or data associated the rolling window or theaggregate estimated number of active clients hash table, such as theaggregate estimated total number of active clients overall and theaggregate estimated number of active clients using mobile computers.

Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 4 is a block diagram that illustrates a computersystem 400 upon which an embodiment of the invention may be implemented.Computer system 400 includes a bus 402 or other communication mechanismfor communicating information, and a hardware processor 404 coupled withbus 402 for processing information. Hardware processor 404 may be, forexample, a general purpose microprocessor.

Computer system 400 also includes a main memory 406, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 402for storing information and instructions to be executed by processor404. Main memory 406 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 404. Such instructions, when stored innon-transitory storage media accessible to processor 404, rendercomputer system 400 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 400 further includes a read only memory (ROM) 408 orother static storage device coupled to bus 402 for storing staticinformation and instructions for processor 404. A storage device 410,such as a magnetic disk or optical disk, is provided and coupled to bus402 for storing information and instructions.

Computer system 400 may be coupled via bus 402 to a display 412, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 414, including alphanumeric and other keys, is coupledto bus 402 for communicating information and command selections toprocessor 404. Another type of user input device is cursor control 416,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 404 and forcontrolling cursor movement on display 412. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 400 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 400 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 400 in response to processor 404 executing one or more sequencesof one or more instructions contained in main memory 406. Suchinstructions may be read into main memory 406 from another storagemedium, such as storage device 410. Execution of the sequences ofinstructions contained in main memory 406 causes processor 404 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 410.Volatile media includes dynamic memory, such as main memory 406. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 402. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 404 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 400 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 402. Bus 402 carries the data tomain memory 406, from which processor 404 retrieves and executes theinstructions. The instructions received by main memory 406 mayoptionally be stored on storage device 410 either before or afterexecution by processor 404.

Computer system 400 also includes a communication interface 418 coupledto bus 402. Communication interface 418 provides a two-way datacommunication coupling to a network link 420 that is connected to alocal network 422. For example, communication interface 418 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 418 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 418sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 420 typically provides data communication through one ormore networks to other data devices. For example, network link 420 mayprovide a connection through local network 422 to a host computer 424 orto data equipment operated by an Internet Service Provider (ISP) 426.ISP 426 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 428. Local network 422 and Internet 428 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 420and through communication interface 418, which carry the digital data toand from computer system 400, are example forms of transmission media.

Computer system 400 can send messages and receive data, includingprogram code, through the network(s), network link 420 and communicationinterface 418. In the Internet example, a server 430 might transmit arequested code for an application program through Internet 428, ISP 426,local network 422 and communication interface 418.

The received code may be executed by processor 404 as it is received,and/or stored in storage device 410, or other non-volatile storage forlater execution.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

What is claimed is:
 1. A system comprising: a memory; one or moreprocessors coupled to the memory and configured to: receive, from aplurality of client computers, a first plurality of heartbeat messages;wherein each heartbeat message of the first plurality of heartbeatmessages: is sent by a client computer of the plurality of clientcomputers, and includes a delta value that indicates an amount of timethat has elapsed since a previous heartbeat message sent from the clientcomputer; determine a first total amount of time based, at least inpart, on the delta value from each heartbeat message of the firstplurality of heartbeat messages; determine a total number of activeclient computers based, at least in part, on the first total amount oftime and a defined amount of time in a first window.
 2. The system ofclaim 1, wherein each heartbeat message of the one or more heartbeatmessages does not include a client identifier.
 3. The system of claim 1,wherein: a first heartbeat message of the first plurality of heartbeatmessages includes a first delta value; a second heartbeat message of thefirst plurality of heartbeat messages includes a second delta value thatis different than the first delta value.
 4. The system of claim 1,wherein: each heartbeat message in a first subset of heartbeat messagesof the first plurality of heartbeat messages includes a first attribute;each heartbeat message in a second subset of heartbeat messages of thefirst plurality of heartbeat messages includes a second attribute; theone or more processors are further configured to: determine a firstsubtotal based, at least in part, on the delta value in each heartbeatmessage of the first subset of heartbeat messages; determine a secondsubtotal based, at least in part, on the delta value in each heartbeatmessage of the second subset of heartbeat messages; receive a firstquery that identifies the first attribute, and in response, determine afirst total number of active client computers based, at least in part,on the first subtotal and the defined amount of time in the firstwindow, but not the second subtotal; receive a second query thatidentifies the second attribute, and in response, determine a secondtotal number of active client computers based, at least in part, on thesecond subtotal and the defined amount of time in the first window, butnot the first subtotal.
 5. The system of claim 1, wherein: eachheartbeat message in a first subset of heartbeat messages of the firstplurality of heartbeat messages includes a first attribute; eachheartbeat message in a second subset of heartbeat messages of the firstplurality of heartbeat messages includes a second attribute; the one ormore processors are further configured to: determine a first subtotalbased, at least in part, on the delta value in each heartbeat message ofthe first subset of heartbeat messages; determine a second subtotalbased, at least in part, on the delta value in each heartbeat message ofthe second subset of heartbeat messages; receive a first query thatidentifies the first attribute and the second attribute, and inresponse, determine a first total number of active client computersbased, at least in part, on the first subtotal, the second subtotal, andthe defined amount of time in the first window.
 6. The system of claim1, wherein the one or more processors are further configured to: send,to one or more first client computers of the plurality of clientcomputers, a first frequency; receive, from the one or more first clientcomputers, a second plurality of heartbeat messages according to thefirst frequency.
 7. The system of claim 6, wherein the one or moreprocessors are further configured to: determine a second frequencybased, at least in part, on one or more factors, wherein the secondfrequency is different than the first frequency; send, to one or moresecond client computers of the plurality of client computers, the secondfrequency; receive, from the one or more second client computers, asecond plurality of heartbeat messages according to the secondfrequency.
 8. The system of claim 6, wherein the one or more processorsare further configured to: determine a load on one or more servercomputers; determine the load exceeds a threshold, and in response:send, to one or more second client computers of the plurality of clientcomputers, a second frequency that is less than the first frequencybased on one or more factors; receive, from the one or more secondclient computers, a second plurality of heartbeat messages according tothe second frequency.
 9. The system of claim 1 comprising a plurality ofserver computers that comprise the one or more processors and thememory, wherein: the first plurality of heartbeat messages arecollectively received by the plurality of server computers; each servercomputer of the plurality of server computers is configured to:determine a subtotal amount of time based, at least in part, on thedelta value from each heartbeat message received by the server computer;send the subtotal amount of time to a particular server computer; theparticular server computer is configured to determine the first totalamount of time based, at least in part, on the subtotal amount of timereceived from each server computer of the plurality of server computers.10. The system of claim 1, wherein the one or more processors arefurther configured to: for each sub-window of a plurality ofsub-windows: receive a plurality of heartbeat messages during thesub-window; determine a subtotal amount of time based, at least in part,on the delta value from each heartbeat message of the plurality ofheartbeat messages received during the sub-window; determine the firstwindow comprises a first set of sub-windows of the plurality ofsub-windows; wherein determining the first total amount of time isbased, at least in part, on the subtotal amount of time of eachsub-window in the first set of sub-windows; wherein the total number ofactive client computers is a first total number of active clientcomputers; determine a second window comprises a second set ofsub-windows of the plurality of sub-windows; determine a second totalamount of time based, at least in part, on the subtotal amount of timeof each sub-window in the second set of sub-windows; determine a secondtotal number of active client computers based, at least in part, on thesecond total amount of time and a defined amount of time in the secondwindow.
 11. The system of claim 10, wherein: the first window comprisesa first sub-window of the plurality the plurality of sub-windows and asecond sub-window of the plurality the plurality of sub-windows, but nota third sub-window of the plurality the plurality of sub-windows; thesecond window comprises the second sub-window of the plurality theplurality of sub-windows and the third sub-window of the plurality theplurality of sub-windows, but not the first sub-window of the pluralitythe plurality of sub-window.
 12. The system of claim 1, wherein thedelta value in a first heartbeat message of the first plurality ofheartbeat messages is different than the delta value in a secondheartbeat message of the first plurality of heartbeat messages.
 13. Amethod comprising: receiving, from a plurality of client computers, afirst plurality of heartbeat messages; wherein each heartbeat message ofthe first plurality of heartbeat messages: is sent by a client computerof the plurality of client computers, and includes a delta value thatindicates an amount of time that has elapsed since a previous heartbeatmessage sent from the client computer; determining a first total amountof time based, at least in part, on the delta value from each heartbeatmessage of the first plurality of heartbeat messages; determining atotal number of active client computers based, at least in part, on thefirst total amount of time and a defined amount of time in a firstwindow; wherein the method is performed by one or more processors. 14.The method of claim 13, wherein each heartbeat message of the one ormore heartbeat messages does not include a client identifier.
 15. Themethod of claim 13, wherein: a first heartbeat message of the firstplurality of heartbeat messages includes a first delta value; a secondheartbeat message of the first plurality of heartbeat messages includesa second delta value that is different than the first delta value. 16.The method of claim 13 further comprising: wherein each heartbeatmessage in a first subset of heartbeat messages of the first pluralityof heartbeat messages includes a first attribute; determining a firstsubtotal based, at least in part, on the delta value in each heartbeatmessage of the first subset of heartbeat messages; wherein eachheartbeat message in a second subset of heartbeat messages of the firstplurality of heartbeat messages includes a second attribute; determininga second subtotal based, at least in part, on the delta value in eachheartbeat message of the second subset of heartbeat messages; receivinga first query that identifies the first attribute, and in response,determining a first total number of active client computers based, atleast in part, on the first subtotal and the defined amount of time inthe first window, but not the second subtotal; receiving a second querythat identifies the second attribute, and in response, determining asecond total number of active client computers based, at least in part,on the second subtotal and the defined amount of time in the firstwindow, but not the first subtotal.
 17. The method of claim 13 furthercomprising: wherein each heartbeat message in a first subset ofheartbeat messages of the first plurality of heartbeat messages includesa first attribute; determining a first subtotal based, at least in part,on the delta value in each heartbeat message of the first subset ofheartbeat messages; wherein each heartbeat message in a second subset ofheartbeat messages of the first plurality of heartbeat messages includesa second attribute; determining a second subtotal based, at least inpart, on the delta value in each heartbeat message of the second subsetof heartbeat messages; receiving a first query that identifies the firstattribute and the second attribute, and in response, determining a firsttotal number of active client computers based, at least in part, on thefirst subtotal, the second subtotal, and the defined amount of time inthe first window.
 18. The method of claim 13 further comprising:sending, to one or more first client computers of the plurality ofclient computers, a first frequency; receiving, from the one or morefirst client computers, a second plurality of heartbeat messagesaccording to the first frequency.
 19. The method of claim 18 furthercomprising: determining a second frequency based, at least in part, onone or more factors, wherein the second frequency is different than thefirst frequency; sending, to one or more second client computers of theplurality of client computers, the second frequency; receiving, from theone or more second client computers, a second plurality of heartbeatmessages according to the second frequency.
 20. The method of claim 18further comprising: determining a load on one or more server computers;determining the load exceeds a threshold, and in response: sending, toone or more second client computers of the plurality of clientcomputers, a second frequency that is less than the first frequencybased on one or more factors; receiving, from the one or more secondclient computers, a second plurality of heartbeat messages according tothe second frequency.
 21. The method of claim 13, wherein the firstplurality of heartbeat messages are collectively received by a pluralityof server computers, and the method further comprising: each servercomputer of the plurality of server computers: determining a subtotalamount of time based, at least in part, on the delta value from eachheartbeat message received by the server computer; sending the subtotalamount of time to a particular server computer; wherein the particularserver computer determines the first total amount of time based, atleast in part, on the subtotal amount of time received from each servercomputer of the plurality of server computers.
 22. The method of claim13 comprising: for each sub-window of a plurality of sub-windows:receiving a plurality of heartbeat messages during the sub-window;determining a subtotal amount of time based, at least in part, on thedelta value from each heartbeat message of the plurality of heartbeatmessages received during the sub-window; determining the first windowcomprises a first set of sub-windows of the plurality of sub-windows;wherein determining the first total amount of time is based, at least inpart, on the subtotal amount of time of each sub-window in the first setof sub-windows; wherein the total number of active client computers is afirst total number of active client computers; determining a secondwindow comprises a second set of sub-windows of the plurality ofsub-windows; determining a second total amount of time based, at leastin part, on the subtotal amount of time of each sub-window in the secondset of sub-windows; determining a second total number of active clientcomputers based, at least in part, on the second total amount of timeand a defined amount of time in the second window.
 23. The method ofclaim 22, wherein: the first window comprises a first sub-window of theplurality the plurality of sub-windows and a second sub-window of theplurality the plurality of sub-windows, but not a third sub-window ofthe plurality the plurality of sub-windows; the second window comprisesthe second sub-window of the plurality the plurality of sub-windows andthe third sub-window of the plurality the plurality of sub-windows, butnot the first sub-window of the plurality the plurality of sub-window.24. The method of claim 13, wherein the delta value in a first heartbeatmessage of the first plurality of heartbeat messages is different thanthe delta value in a second heartbeat message of the first plurality ofheartbeat messages.